

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>crimson &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/underscore.js"></script>
        <script src="../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="error handling" href="../error-handling/" />
    <link rel="prev" title="Crimson developer documentation" href="../" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../internals/">Ceph 内幕</a> &raquo;</li>
        
          <li><a href="../">Crimson developer documentation</a> &raquo;</li>
        
      <li>crimson</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../_sources/dev/crimson/crimson.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../">
          

          
            
            <img src="../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../blkin/">Tracing Ceph With LTTng</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../blkin/#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cache-pool/">Cache pool</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../context/">CephContext</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../continuous-integration/">Continuous Integration Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dev_cluster_deployement/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dev_cluster_deployement/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../iana/">IANA 号</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">Crimson developer documentation</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="#">crimson</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#building-crimson">Building Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#running-crimson">Running Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#metrics-and-tracing">Metrics and Tracing</a></li>
<li class="toctree-l4"><a class="reference internal" href="#profiling-crimson">Profiling Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#hacking-crimson">Hacking Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#debugging-crimson">Debugging Crimson</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../error-handling/">error handling</a></li>
<li class="toctree-l3"><a class="reference internal" href="../osd/">osd</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pipeline/">The <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> pipeline</a></li>
<li class="toctree-l3"><a class="reference internal" href="../poseidonstore/">PoseidonStore</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="crimson">
<h1>crimson<a class="headerlink" href="#crimson" title="Permalink to this headline">¶</a></h1>
<p>Crimson is the code name of crimson-osd, which is the next generation ceph-osd.
It targets fast networking devices, fast storage devices by leveraging state of
the art technologies like DPDK and SPDK, for better performance. And it will
keep the support of HDDs and low-end SSDs via BlueStore. Crimson will try to
be backward compatible with classic OSD.</p>
<div class="section" id="building-crimson">
<h2>Building Crimson<a class="headerlink" href="#building-crimson" title="Permalink to this headline">¶</a></h2>
<p>Crimson is not enabled by default. To enable it:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nv">WITH_SEASTAR</span><span class="o">=</span><span class="nb">true</span> ./install-deps.sh
<span class="gp">$ </span>mkdir build <span class="o">&amp;&amp;</span> <span class="nb">cd</span> build
<span class="gp">$ </span>cmake -DWITH_SEASTAR<span class="o">=</span>ON ..
</pre></div>
</div>
<p>Please note, <a class="reference external" href="https://github.com/google/sanitizers/wiki/AddressSanitizer">ASan</a> is enabled by default if crimson is built from a source
cloned using git.</p>
</div>
<div class="section" id="running-crimson">
<h2>Running Crimson<a class="headerlink" href="#running-crimson" title="Permalink to this headline">¶</a></h2>
<p>As you might expect, crimson is not featurewise on par with its predecessor yet.</p>
<div class="section" id="object-store-backend">
<h3>object store backend<a class="headerlink" href="#object-store-backend" title="Permalink to this headline">¶</a></h3>
<p>At the moment, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> offers both native and alienized object store
backends. The native object store backends perform IO using seastar reactor.
They are:</p>
<dl class="describe">
<dt>
<code class="sig-name descname"><span class="pre">cyanstore</span></code></dt>
<dd><p>CyanStore is modeled after memstore in classic OSD.</p>
</dd></dl>

<dl class="describe">
<dt>
<code class="sig-name descname"><span class="pre">seastore</span></code></dt>
<dd><p>Seastore is still under active development.</p>
</dd></dl>

<p>While the alienized object store backends are backed by a thread pool, which
is a proxy of the alien store adaptor running in SeaStar. The proxy issues
requests to object stores running in alien threads, i.e., worker threads not
managed by the Seastar framework. They are:</p>
<dl class="describe">
<dt>
<code class="sig-name descname"><span class="pre">memstore</span></code></dt>
<dd><p>The memory backed object store</p>
</dd></dl>

<dl class="describe">
<dt>
<code class="sig-name descname"><span class="pre">bluestore</span></code></dt>
<dd><p>The object store used by classic OSD by default.</p>
</dd></dl>

</div>
<div class="section" id="daemonize">
<h3>daemonize<a class="headerlink" href="#daemonize" title="Permalink to this headline">¶</a></h3>
<p>Unlike <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> does not daemonize itself even if the
<code class="docutils literal notranslate"><span class="pre">daemonize</span></code> option is enabled. Because, to read this option, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>
needs to ready its config sharded service, but this sharded service lives
in the seastar reactor. If we fork a child process and exit the parent after
starting the Seastar engine, that will leave us with a single thread which is
the replica of the thread calls <a class="reference external" href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html">fork()</a>. This would unnecessarily complicate
the code, if we would have tackled this problem in crimson.</p>
<p>Since a lot of GNU/Linux distros are using systemd nowadays, which is able to
daemonize the application, there is no need to daemonize by ourselves. For
those who are using sysvinit, they can use <code class="docutils literal notranslate"><span class="pre">start-stop-daemon</span></code> for daemonizing
<code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>. If this is not acceptable, we can whip up a helper utility
to do the trick.</p>
</div>
<div class="section" id="logging">
<h3>logging<a class="headerlink" href="#logging" title="Permalink to this headline">¶</a></h3>
<p>Currently, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> uses the logging utility offered by Seastar. see
<code class="docutils literal notranslate"><span class="pre">src/common/dout.h</span></code> for the mapping between different logging levels to
the severity levels in Seastar. For instance, the messages sent to <code class="docutils literal notranslate"><span class="pre">derr</span></code>
will be printed using <code class="docutils literal notranslate"><span class="pre">logger::error()</span></code>, and the messages with debug level
over <code class="docutils literal notranslate"><span class="pre">20</span></code> will be printed using <code class="docutils literal notranslate"><span class="pre">logger::trace()</span></code>.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<tbody>
<tr class="row-odd"><td><p>ceph</p></td>
<td><p>seastar</p></td>
</tr>
<tr class="row-even"><td><p>&lt; 0</p></td>
<td><p>error</p></td>
</tr>
<tr class="row-odd"><td><p>0</p></td>
<td><p>warn</p></td>
</tr>
<tr class="row-even"><td><p>[1, 6)</p></td>
<td><p>info</p></td>
</tr>
<tr class="row-odd"><td><p>[6, 20]</p></td>
<td><p>debug</p></td>
</tr>
<tr class="row-even"><td><p>&gt;  20</p></td>
<td><p>trace</p></td>
</tr>
</tbody>
</table>
<p>Please note, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>
does not send the logging message to specified <code class="docutils literal notranslate"><span class="pre">log_file</span></code>. It writes
the logging messages to stdout and/or syslog. Again, this behavior can be
changed using <code class="docutils literal notranslate"><span class="pre">--log-to-stdout</span></code> and <code class="docutils literal notranslate"><span class="pre">--log-to-syslog</span></code> command line
options. By default, <code class="docutils literal notranslate"><span class="pre">log-to-stdout</span></code> is enabled, and the latter disabled.</p>
</div>
<div class="section" id="vstart-sh">
<h3>vstart.sh<a class="headerlink" href="#vstart-sh" title="Permalink to this headline">¶</a></h3>
<p>To facilitate the development of crimson, following options would be handy when
using <code class="docutils literal notranslate"><span class="pre">vstart.sh</span></code>,</p>
<dl>
<dt><code class="docutils literal notranslate"><span class="pre">--crimson</span></code></dt><dd><p>start <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> instead of <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code></p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--nodaemon</span></code></dt><dd><p>do not daemonize the service</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--redirect-output</span></code></dt><dd><p>redirect the stdout and stderr of service to <code class="docutils literal notranslate"><span class="pre">out/$type.$num.stdout</span></code>.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--osd-args</span></code></dt><dd><p>pass extra command line options to crimson-osd or ceph-osd. It’s quite
useful for passing Seastar options to crimson-osd. For instance, you could
use <code class="docutils literal notranslate"><span class="pre">--osd-args</span> <span class="pre">&quot;--memory</span> <span class="pre">2G&quot;</span></code> to set the memory to use. Please refer
the output of:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">crimson-osd --help-seastar</span>
</pre></div>
</div>
<p>for more Seastar specific command line options.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--cyanstore</span></code></dt><dd><p>use the CyanStore as the object store backend.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--bluestore</span></code></dt><dd><p>use the alienized BlueStore as the object store backend. This is the default
setting, if not specified otherwise.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--memstore</span></code></dt><dd><p>use the alienized MemStore as the object store backend.</p>
</dd>
</dl>
<p>So, a typical command to start a single-crimson-node cluster is:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$  </span><span class="nv">MGR</span><span class="o">=</span><span class="m">1</span> <span class="nv">MON</span><span class="o">=</span><span class="m">1</span> <span class="nv">OSD</span><span class="o">=</span><span class="m">1</span> <span class="nv">MDS</span><span class="o">=</span><span class="m">0</span> <span class="nv">RGW</span><span class="o">=</span><span class="m">0</span> ../src/vstart.sh -n -x <span class="se">\</span>
  --without-dashboard --cyanstore <span class="se">\</span>
  --crimson --redirect-output <span class="se">\</span>
  --osd-args <span class="s2">&quot;--memory 4G&quot;</span>
</pre></div>
</div>
<p>Where we assign 4 GiB memory, a single thread running on core-0 to crimson-osd.</p>
<p>You could stop the vstart cluster using:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>../src/stop.sh --crimson
</pre></div>
</div>
</div>
</div>
<div class="section" id="metrics-and-tracing">
<h2>Metrics and Tracing<a class="headerlink" href="#metrics-and-tracing" title="Permalink to this headline">¶</a></h2>
<p>Crimson offers three ways to report the stats and metrics:</p>
<div class="section" id="pg-stats-reported-to-mgr">
<h3>pg stats reported to mgr<a class="headerlink" href="#pg-stats-reported-to-mgr" title="Permalink to this headline">¶</a></h3>
<p>Crimson collects the per-pg, per-pool, and per-osd stats in a <cite>MPGStats</cite>
message, and send it over to mgr, so that the mgr modules can query
them using the <cite>MgrModule.get()</cite> method.</p>
</div>
<div class="section" id="asock-command">
<h3>asock command<a class="headerlink" href="#asock-command" title="Permalink to this headline">¶</a></h3>
<p>an asock command is offered for dumping the metrics:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ceph tell osd.0 dump_metrics
<span class="gp">$ </span>ceph tell osd.0 dump_metrics reactor_utilization
</pre></div>
</div>
<p>Where <cite>reactor_utilization</cite> is an optional string allowing us to filter
the dumped metrics by prefix.</p>
</div>
<div class="section" id="prometheus-text-protocol">
<h3>Prometheus text protocol<a class="headerlink" href="#prometheus-text-protocol" title="Permalink to this headline">¶</a></h3>
<p>the listening port and address can be configured using the command line options of
<cite>–prometheus_port</cite>
see <a class="reference external" href="https://github.com/scylladb/seastar/blob/master/doc/prometheus.md">Prometheus</a> for more details.</p>
</div>
</div>
<div class="section" id="profiling-crimson">
<h2>Profiling Crimson<a class="headerlink" href="#profiling-crimson" title="Permalink to this headline">¶</a></h2>
<div class="section" id="fio">
<h3>fio<a class="headerlink" href="#fio" title="Permalink to this headline">¶</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code> exposes configurable <code class="docutils literal notranslate"><span class="pre">FuturizedStore</span></code> internals as an
NBD server for use with fio.</p>
<p>To use fio to test <code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code>,</p>
<ol class="arabic">
<li><p>You will need to install <code class="docutils literal notranslate"><span class="pre">libnbd</span></code>, and compile fio like</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">apt-get install libnbd-dev</span>
<span class="prompt1">git clone git://git.kernel.dk/fio.git</span>
<span class="prompt1"><span class="nb">cd</span> fio</span>
<span class="prompt1">./configure --enable-libnbd</span>
<span class="prompt1">make</span>
</pre></div></div></li>
<li><p>Build <code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="nb">cd</span> build</span>
<span class="prompt1">ninja crimson-store-nbd</span>
</pre></div></div></li>
<li><p>Run the <code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code> server with a block device. Please specify
the path to the raw device, like <code class="docutils literal notranslate"><span class="pre">/dev/nvme1n1</span></code> in place of the created
file for testing with a block device.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="nb">export</span> <span class="nv">disk_img</span><span class="o">=</span>/tmp/disk.img</span>
<span class="prompt1"><span class="nb">export</span> <span class="nv">unix_socket</span><span class="o">=</span>/tmp/store_nbd_socket.sock</span>
<span class="prompt1">rm -f <span class="nv">$disk_img</span> <span class="nv">$unix_socket</span></span>
<span class="prompt1">truncate -s 512M <span class="nv">$disk_img</span></span>
<span class="prompt1">./bin/crimson-store-nbd <span class="se">\</span>
  --device-path <span class="nv">$disk_img</span> <span class="se">\</span>
  --smp <span class="m">1</span> <span class="se">\</span>
  --mkfs <span class="nb">true</span> <span class="se">\</span>
  --type transaction_manager <span class="se">\</span>
  --uds-path <span class="si">${</span><span class="nv">unix_socket</span><span class="si">}</span> <span class="p">&amp;</span></span>
</pre></div></div><p>in which,</p>
<dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">--smp</span></code></dt><dd><p>how many CPU cores are used</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--mkfs</span></code></dt><dd><p>initialize the device first</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--type</span></code></dt><dd><p>which backend to use. If <code class="docutils literal notranslate"><span class="pre">transaction_manager</span></code> is specified, SeaStore’s
<code class="docutils literal notranslate"><span class="pre">TransactionManager</span></code> and <code class="docutils literal notranslate"><span class="pre">BlockSegmentManager</span></code> are used to emulate a
block device. Otherwise, this option is used to choose a backend of
<code class="docutils literal notranslate"><span class="pre">FuturizedStore</span></code>, where the whole “device” is divided into multiple
fixed-size objects whose size is specified by <code class="docutils literal notranslate"><span class="pre">--object-size</span></code>. So, if
you are only interested in testing the lower-level implementation of
SeaStore like logical address translation layer and garbage collection
without the object store semantics, <code class="docutils literal notranslate"><span class="pre">transaction_manager</span></code> would be a
better choice.</p>
</dd>
</dl>
</li>
<li><p>Create an fio job file named <code class="docutils literal notranslate"><span class="pre">nbd.fio</span></code></p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="k">[global]</span>
<span class="na">ioengine</span><span class="o">=</span><span class="s">nbd</span>
<span class="na">uri</span><span class="o">=</span><span class="s">nbd+unix:///?socket=${unix_socket}</span>
<span class="na">rw</span><span class="o">=</span><span class="s">randrw</span>
<span class="na">time_based</span>
<span class="na">runtime</span><span class="o">=</span><span class="s">120</span>
<span class="na">group_reporting</span>
<span class="na">iodepth</span><span class="o">=</span><span class="s">1</span>
<span class="na">size</span><span class="o">=</span><span class="s">512M</span>

<span class="k">[job0]</span>
<span class="na">offset</span><span class="o">=</span><span class="s">0</span>
</pre></div>
</div>
</li>
<li><p>Test the crimson object store using the fio compiled just now</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">./fio nbd.fio</span>
</pre></div></div></li>
</ol>
</div>
<div class="section" id="cbt">
<h3>CBT<a class="headerlink" href="#cbt" title="Permalink to this headline">¶</a></h3>
<p>We can use <a class="reference external" href="https://github.com/ceph/cbt">cbt</a> for performing perf tests:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>git checkout master
<span class="gp">$ </span>make crimson-osd
<span class="gp">$ </span>../src/script/run-cbt.sh --cbt ~/dev/cbt -a /tmp/baseline ../src/test/crimson/cbt/radosbench_4K_read.yaml
<span class="gp">$ </span>git checkout yet-another-pr
<span class="gp">$ </span>make crimson-osd
<span class="gp">$ </span>../src/script/run-cbt.sh --cbt ~/dev/cbt -a /tmp/yap ../src/test/crimson/cbt/radosbench_4K_read.yaml
<span class="gp">$ </span>~/dev/cbt/compare.py -b /tmp/baseline -a /tmp/yap -v
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/0: bandwidth: (or (greater) (near 0.05)):: 0.183165/0.186155  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/0: iops_avg: (or (greater) (near 0.05)):: 46.0/47.0  =&gt; accepted</span>
<span class="go">19:48:23 - WARNING  - cbt      - prefill/gen8/0: iops_stddev: (or (less) (near 0.05)):: 10.4403/6.65833  =&gt; rejected</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.340868/0.333712  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: bandwidth: (or (greater) (near 0.05)):: 0.190447/0.177619  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_avg: (or (greater) (near 0.05)):: 48.0/45.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_stddev: (or (less) (near 0.05)):: 6.1101/9.81495  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.325163/0.350251  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: bandwidth: (or (greater) (near 0.05)):: 1.24654/1.22336  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: iops_avg: (or (greater) (near 0.05)):: 319.0/313.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.0497733/0.0509029  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: bandwidth: (or (greater) (near 0.05)):: 1.22717/1.11372  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: iops_avg: (or (greater) (near 0.05)):: 314.0/285.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.0508262/0.0557337  =&gt; accepted</span>
<span class="go">19:48:23 - WARNING  - cbt      - 1 tests failed out of 16</span>
</pre></div>
</div>
<p>Where we compile and run the same test against two branches. One is <code class="docutils literal notranslate"><span class="pre">master</span></code>, another is <code class="docutils literal notranslate"><span class="pre">yet-another-pr</span></code> branch.
And then we compare the test results. Along with every test case, a set of rules is defined to check if we have
performance regressions when comparing two set of test results. If a possible regression is found, the rule and
corresponding test results are highlighted.</p>
</div>
</div>
<div class="section" id="hacking-crimson">
<h2>Hacking Crimson<a class="headerlink" href="#hacking-crimson" title="Permalink to this headline">¶</a></h2>
<div class="section" id="seastar-documents">
<h3>Seastar Documents<a class="headerlink" href="#seastar-documents" title="Permalink to this headline">¶</a></h3>
<p>See <a class="reference external" href="https://github.com/scylladb/seastar/blob/master/doc/tutorial.md">Seastar Tutorial</a> .
Or build a browsable version and start an HTTP server:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nb">cd</span> seastar
<span class="gp">$ </span>./configure.py --mode debug
<span class="gp">$ </span>ninja -C build/debug docs
<span class="gp">$ </span>python3 -m http.server -d build/debug/doc/html
</pre></div>
</div>
<p>You might want to install <code class="docutils literal notranslate"><span class="pre">pandoc</span></code> and other dependencies beforehand.</p>
</div>
</div>
<div class="section" id="debugging-crimson">
<h2>Debugging Crimson<a class="headerlink" href="#debugging-crimson" title="Permalink to this headline">¶</a></h2>
<div class="section" id="debugging-with-gdb">
<h3>Debugging with GDB<a class="headerlink" href="#debugging-with-gdb" title="Permalink to this headline">¶</a></h3>
<p>The <a class="reference external" href="https://github.com/scylladb/scylla/blob/master/docs/guides/debugging.md#tips-and-tricks">tips</a> for debugging Scylla also apply to Crimson.</p>
</div>
<div class="section" id="human-readable-backtraces-with-addr2line">
<h3>Human-readable backtraces with addr2line<a class="headerlink" href="#human-readable-backtraces-with-addr2line" title="Permalink to this headline">¶</a></h3>
<p>When a seastar application crashes, it leaves us with a serial of addresses, like:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">Segmentation fault.</span>
<span class="go">Backtrace:</span>
<span class="go">  0x00000000108254aa</span>
<span class="go">  0x00000000107f74b9</span>
<span class="go">  0x00000000105366cc</span>
<span class="go">  0x000000001053682c</span>
<span class="go">  0x00000000105d2c2e</span>
<span class="go">  0x0000000010629b96</span>
<span class="go">  0x0000000010629c31</span>
<span class="go">  0x00002a02ebd8272f</span>
<span class="go">  0x00000000105d93ee</span>
<span class="go">  0x00000000103eff59</span>
<span class="go">  0x000000000d9c1d0a</span>
<span class="go">  /lib/x86_64-linux-gnu/libc.so.6+0x000000000002409a</span>
<span class="go">  0x000000000d833ac9</span>
<span class="go">Segmentation fault</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">seastar-addr2line</span></code> offered by Seastar can be used to decipher these
addresses. After running the script, it will be waiting for input from stdin,
so we need to copy and paste the above addresses, then send the EOF by inputting
<code class="docutils literal notranslate"><span class="pre">control-D</span></code> in the terminal:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>../src/seastar/scripts/seastar-addr2line -e bin/crimson-osd

<span class="go">  0x00000000108254aa</span>
<span class="go">  0x00000000107f74b9</span>
<span class="go">  0x00000000105366cc</span>
<span class="go">  0x000000001053682c</span>
<span class="go">  0x00000000105d2c2e</span>
<span class="go">  0x0000000010629b96</span>
<span class="go">  0x0000000010629c31</span>
<span class="go">  0x00002a02ebd8272f</span>
<span class="go">  0x00000000105d93ee</span>
<span class="go">  0x00000000103eff59</span>
<span class="go">  0x000000000d9c1d0a</span>
<span class="go">  0x00000000108254aa</span>
<span class="go">[Backtrace #0]</span>
<span class="go">seastar::backtrace_buffer::append_backtrace() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1136</span>
<span class="go">seastar::print_with_backtrace(seastar::backtrace_buffer&amp;) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1157</span>
<span class="go">seastar::print_with_backtrace(char const*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1164</span>
<span class="go">seastar::sigsegv_action() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5119</span>
<span class="go">seastar::install_oneshot_signal_handler&lt;11, &amp;seastar::sigsegv_action&gt;()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5105</span>
<span class="go">seastar::install_oneshot_signal_handler&lt;11, &amp;seastar::sigsegv_action&gt;()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5101</span>
<span class="go">?? ??:0</span>
<span class="go">seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5418</span>
<span class="go">seastar::app_template::run_deprecated(int, char**, std::function&lt;void ()&gt;&amp;&amp;) at /home/kefu/dev/ceph/build/../src/seastar/src/core/app-template.cc:173 (discriminator 5)</span>
<span class="go">main at /home/kefu/dev/ceph/build/../src/crimson/osd/main.cc:131 (discriminator 1)</span>
</pre></div>
</div>
<p>Please note, <code class="docutils literal notranslate"><span class="pre">seastar-addr2line</span></code> is able to extract the addresses from
the input, so you can also paste the log messages like:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">2020-07-22T11:37:04.500 INFO:teuthology.orchestra.run.smithi061.stderr:Backtrace:</span>
<span class="go">2020-07-22T11:37:04.500 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e78dbc</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e3e7f0</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e3e8b8</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e3e985</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  /lib64/libpthread.so.0+0x0000000000012dbf</span>
</pre></div>
</div>
<p>Unlike classic OSD, crimson does not print a human-readable backtrace when it
handles fatal signals like <cite>SIGSEGV</cite> or <cite>SIGABRT</cite>. And it is more complicated
when it comes to a stripped binary. So before planting a signal handler for
those signals in crimson, we could to use <cite>script/ceph-debug-docker.sh</cite> to parse
the addresses in the backtrace:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp"># </span>assuming you are under the <span class="nb">source</span> tree of ceph
<span class="gp">$ </span>./src/script/ceph-debug-docker.sh  --flavor crimson master:27e237c137c330ebb82627166927b7681b20d0aa centos:8
<span class="go">....</span>
<span class="gp">[root@3deb50a8ad51 ~]# </span>wget -q https://raw.githubusercontent.com/scylladb/seastar/master/scripts/seastar-addr2line
<span class="gp">[root@3deb50a8ad51 ~]# </span>dnf install -q -y file
<span class="gp">[root@3deb50a8ad51 ~]# </span>python3 seastar-addr2line -e /usr/bin/crimson-osd
<span class="gp"># </span>paste the backtrace here
</pre></div>
</div>
</div>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../error-handling/" class="btn btn-neutral float-right" title="error handling" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../" class="btn btn-neutral float-left" title="Crimson developer documentation" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>